A Comparative Analysis of Chatgpt Vs Expert in Managing Anticancer Drug in Patients Renal Insufficiency

Introduction

The appropriate and safe use of anticancer drugs in patients with renal insufficiency (RI) remains a significant challenge in oncology. Recent advancements in natural language processing and machine learning have introduced novel possibilities for clinical decision support. ChatGPT, a state-of-the-art language model, presents a potential solution to support healthcare professionals (HCPs) in making informed treatment management for patients with RI. But a question remains, how ChatGPT would perform is such situations? The aim of this analysis was to compare the recommendations from (human) experts vs ChatGPT regarding the management of anticancer drug in RI patients and more specifically about the potential dose adjustment.

Methods

In this work, we aimed to compare the performance of ChatGPT 3.5 with experts that are making recommendations on SiteGPR.com. SiteGPR is providing recommendations on how to manage drugs in RI patients and is led by “Service ICAR” from the Nephrology department of the French Pitié-Salpêtrière Hospital (Paris, France). The prompt, “Is there a need to adapt the dose of [name of the drug] in patients with renal impairment and if yes for which level of renal function? was run in a separate ”new chat“ for 175 anticancer drugs in March 2023. Answers were collected and compared with the recommendations from SiteGPR. Both answers from ChatGPT and recommendations from SiteGPR were categorised as ”Yes“ when dose adjustment was required, ”No“ when adjustment was not necessary, and ”ND“ when no data were available. For both ”Yes“ and ”ND“, the level of renal function for which there is a need to change the dose (”Yes“) and for which there are no data below (”ND“) was also collected

Results

Among 175 anticancer drugs, ChatGPT and SiteGPR had the same general conclusion for 37.1% (65/175) of these drugs (Table, Bold) when the level of RI was not analysed. When ChatGPT had a recommendation to adapt (Yes) or not to adapt (No) the dose of an anticancer drug (n=77 Yes or No drugs), SiteGPR did not agree for 57.1% (44/77) of the drugs. Reversely, when SiteGPR had a strong recommendation to adapt (Yes) or not to adapt (No) the dose of an anticancer drug (n=118 Yes or No drugs), ChatGPT was not aligned in 72.0% (85/118) of the cases.

From a clinical perspective, if a HCP had followed ChatGPT instead SiteGPR, patients would have been exposed to a misused for 48.6% (85/175) of the drugs. Both ChatGPR and SiteGPR were not aware of few drugs, however, there is always a way to contact “Service ICAR” if a drug was not investigated on the website.

Regarding the level of renal function for which there was a need to adapt the dose (Yes), ChatGPT was able to give a cut-off for 9.6% (7/73) of the anticancer drugs, vs 100.0% (42/42) for SiteGPR. Finally, from a quality perspective, it is important to mention that ChatGPT answers were not always clear. Regarding the level of renal function for which there was no data (ND), ChatGPT was able to give a cut-off for 12.0% (11/92) of the anticancer drugs, vs 100.0% (44/44) for SiteGPR. Finally, from a quality perspective, it is important to mention that ChatGPT answers were not always clear.

At the end, when comparing the general recommendations and when considering the level of renal function and after excluding all N/A drugs, ChatGPT and SiteGPR had the same conclusion for only 5.6% (9/161) of anticancer drugs.

Conclusion

Our findings suggest that ChatGPT does not have (yet) the potential to be an effective clinical decision support tool for optimizing anticancer drug management in RI patients. Further investigations will compare the recommended dose and monitoring between ChatGPT 4.0 and SiteGPR and will also include other AI models. Combining the power of artificial intelligence with human expertise can lead to more personalized and evidence-based treatment decisions in cancer patients. However, it is crucial to acknowledge that while ChatGPT shows some promise, human clinicians remain indispensable for addressing intricate and unique patient situations.